Be surprised in an embarrassing or guilty posture, as in We spent a lot of time preparing for the inspection; we didn't want to get caught with our pants down. This phrase presumably alludes to someone's pants being lowered to attend to bathroom needs but is not considered particularly vulgar. It is similar to